Controlling a virtual body by thought in a highly-immersive virtual environment - a case study in using a brain-computer interface in a virtual-reality cave-like system

A brain-computer interface (BCI) can arguably be considered the ultimate user interface, where humans operate a computer using thought alone. We have integrated the Graz-BCI into a highly immersive Cave-like system. In this paper we report a case study where three participants were able to control their avatar using only their thought. We have analyzed the participants’ subjective experience using an in-depth qualitative methodology. We also discuss some limitations of BCI in controlling a virtual environment, and interaction design decisions that needed to be made. Brain-computer interface (BCI) has been studied extensively as a tool for paralyzed patients, which may augment their communication with the external world and allow them better control of their limbs. However, once it has been developed for these critical applications, we expect it will have profound implications on many other types of user interfaces and applications. BCI could be one of the most significant steps following “direct manipulation interfaces” (Schneiderman, 1983) – where intention is mapped directly into interaction, rather than being conveyed through motor movements. Furthermore, if used in an immersive virtual environment (IVE) this could be a completely novel experience and, in the future, lead to unprecedented levels in the sense of presence (for recent reviews of the concept of presence see (Vives and Slater, 2005) and (Riva et al., 2003)). A key requirement for a successful experience in an immersive virtual environment (IVE) is the representation of the participant, or its avatar (Pandzic et al., 1997; Slater et al., 1994; Slater et al., 1998). This paper describes the first ever study where participants control their own avatar using only their thoughts. Three subjects were able to use the GrazBCI to control an avatar, and their subjective experience was assessed using questionnaires and a semistructured interview. Naturally, a third-person avatar, such as used in this experiment, is only one possible interface to an IVE. Using a BCI to control an IVE by thought raises several major human-computer interaction (HCI) issues: whether classification of thought patterns is continuous (asynchronous BCI) or only takes place in specific moments (synchronous BCI), the number of input classes recognized, the importance of feedback, and the nature of the mapping between thoughts and resulting action in the IVE. In this paper we refer to these issues, and present a case study that specifically addresses the issues of feedback and mapping. A critical initial hypothesis is that natural mapping between thought processes and IVE functionality would improve the experience. A one-to-one mapping seemingly makes intuitive sense, but having this mapping is constraining because we are limited in the scope of thought patterns that we can detect based on contemporary brain recording techniques. In addition, it precludes other more complex or more fanciful body image mappings; what if we want to experiment with lobster avatars? (See Jaron Lanier’s “everyone can be a lobster” statement in http://www.edge.org/q2006/q06 7.html#lanier). In the case study reported here we have found out that natural mapping was reported to feel more natural and easy than when the mapping was reversed. However, the results do not indicate that BCI accuracy was better with natural mapping than with reversed mapping. The main implication of our case study is that this new type of interface, whereby IVE participants control their avatars by thought, is possible, and should be further pursued. In addition, we reveal new insights about the HCI issues that are involved in such an interface, and provide a first glance into what the experience of using such an interface may be like.